Tag: gradient noise scale

Global batch size and convergence in large language model training

How does global batch size affect the convergence and compute efficiency of LLM pretraining?

The global batch size (tokens per optimizer step) does not change where training converges so much as how efficiently it gets there: it trades steps against tokens against wall-clock. A measurable quantity, the gradient noise scale, marks the critical batch size beyond which larger batches buy almost no extra speed, and the evidence from both theory and production LLM runs says the critical batch size grows with training progress and with dataset size, not model size. Learning-rate scaling rules (linear for SGD, subtler for Adam) are what make large batches safe, and batch ramping is now standard practice across GPT-3, PaLM, Llama 3, and DeepSeek-V3.

batch size · LLM pretraining · gradient noise scale · critical batch size · convergence